In recent years, development of an AR (Augmented Reality) technique has been advanced in which a virtual object is superimposed onto an image of a real world and the resultant image is displayed, whereby the virtual object appears to be present in the real world.
For example, a stereoscopic display apparatus disclosed in Patent Literature 1 (Japanese Laid-Open Patent Publication No. 2008-146109) calculates the relative positions and relative orientations of a camera for a right eye and a camera for a left eye with respect to a marker placed in the real world, from respective images shot by the camera for a right eye and the camera for a left eye, which are attached to a head mounted display, and then, based on the result of the calculation, generates an image of a virtual object for a right eye and an image for the virtual object for a left eye. Then, the stereoscopic display apparatus superimposes the image of the virtual object for a right eye and the image for the virtual object for a left eye onto the images shot by the camera for a right eye and the camera for a left eye, respectively, and displays the resultant composite images on an LCD (liquid crystal display) for a right eye and an LCD for a left eye, respectively.
In addition, Non-Patent Literature 1 (Hirokazu Kato, Mark Billinghurst, “Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System,” iwar, pp. 85, 2nd IEEE and ACM International Workshop on Augmented Reality, 1999) discloses a method of, based on the position and the orientation of a marker in an image shot by a camera, calculating the relative positions and the relative orientations of the marker and the camera with respect to each other in the real world.
However, in the case where the relative positions and the relative orientations of the marker and the camera with respect to each other in the real world are calculated based on the position and the orientation of the marker in an image shot by the camera, if the image shot by the camera is blurred or if the accuracy of marker recognition is not perfect, error occurs in a result of calculation. Therefore, in the stereoscopic display apparatus disclosed in Patent Literature 1, the relative position and orientation of the camera for a right eye with respect to the marker calculated based on an image shot by the camera for a right eye are not always accurate, and also, the relative position and orientation of the camera for a left eye with respect to the marker calculated based on an image shot by the camera for a left eye are not always accurate. An image for a right eye including a virtual object, and an image for a left eye including the virtual object that are generated based on a result of calculation that is not accurate, do not have consistency with each other, that is, they are inconsistence with each other. As a result, a user cannot normally stereoscopically view the virtual object.
Therefore, an object of the present invention is to provide a computer-readable storage medium, an image display apparatus, an image display system, and an image display method that are capable of displaying a virtual object so as to be normally stereoscopically visible.
The present invention has the following features to attain the object mentioned above.
A computer-readable storage medium of the present invention is a computer-readable storage medium having stored therein an image display program which stereoscopically displays a real world image onto which a 3-dimensional virtual object is superimposed, on a screen of a stereoscopic display apparatus capable of providing a stereoscopic view, by using outputs from a real camera imaging section for a right eye and a real camera imaging section for a left eye, the image display program causing a computer to function as: position orientation calculation means for recognizing a predetermined shooting target in each of pieces of real world image data respectively outputted from the two real camera imaging sections, and calculating position orientation information indicating relative positions and relative orientations of each of the two real camera imaging sections, and the predetermined shooting target with respect to each other; position orientation integrating means for integrating the two pieces of position orientation information calculated by the position orientation calculation means, thereby calculating one piece of position orientation information; virtual camera setting means for determining the position and the orientation of a right virtual camera for generating a virtual image for a right eye, and the position and the orientation of a left virtual camera for generating a virtual image for a left eye, in a predetermined virtual space, based on the one piece of position orientation information integrated by the first position orientation integrating means; right virtual space image generation means for generating a right virtual space image indicating the predetermined virtual space as it is looked at from the right virtual camera; left virtual space image generation means for generating a left virtual space image indicating the predetermined virtual space as it is looked at from the left virtual camera; and display control means for superimposing the right virtual space image onto the piece of real world image data outputted from the real camera imaging section for a right eye, and superimposing the left virtual space image onto the piece of real world image data outputted from the real camera imaging section for a left eye, to output images for a stereoscopic view to the stereoscopic display apparatus.
In the present invention, the virtual camera setting means may determine the positions and the orientations of the right virtual camera and the left virtual camera such that the relationship between relative orientations of the right virtual camera and the left virtual camera with respect to each other is the same as the relationship between designed relative orientations of the real camera imaging section for a right eye and the real camera imaging section for a left eye with respect to each other.
In the present invention, the position orientation integrating means may integrate the information about the relative orientations of the real camera imaging section for a right eye and the predetermined shooting target with respect to each other, and the information about the relative orientations of the real camera imaging section for a left eye and the predetermined shooting target with respect to each other, which are calculated by the position orientation calculation means, thereby calculating one orientation. The virtual camera setting means may use the one orientation integrated by the position orientation integrating means, as each of the orientations of the right virtual camera and the left virtual camera.
In the present invention, the position orientation integrating means may integrate the information about the relative positions of the real camera imaging section for a right eye and the predetermined shooting target with respect to each other, and the information about the relative positions of the real camera imaging section for a left eye and the predetermined shooting target with respect to each other, which are calculated by the position orientation calculation means, thereby calculating one position. The virtual camera setting means may set the positions of the right virtual camera and the left virtual camera so as to be symmetric about the one position integrated by the position orientation integrating means.
In the present invention, the position orientation integrating means may average the information about the relative orientations of the real camera imaging section for a right eye and the predetermined shooting target with respect to each other, and the information about the relative orientations of the real camera imaging section for a left eye and the predetermined shooting target with respect to each other, which are calculated by the position orientation calculation means, thereby calculating an averaged orientation. The virtual camera setting means may use the averaged orientation calculated by the position orientation integrating means, as each of the orientations of the right virtual camera and the left virtual camera.
In the present invention, the position orientation integrating means may calculate the midpoint between the relative position of the real camera imaging section for a right eye and the relative position of the real camera imaging section for a left eye, by using the information about the relative positions of the real camera imaging section for a right eye and the predetermined shooting target with respect to each other, and the information about the relative positions of the real camera imaging section for a left eye and the predetermined shooting target with respect to each other, which are calculated by the position orientation calculation means. The virtual camera setting means may set the positions of the right virtual camera and the left virtual camera so as to be symmetric about the midpoint calculated by the position orientation integrating means.
In the present invention, the image display program may further cause the computer to function as virtual cameras distance determination means for determining the distance between the right virtual camera and the left virtual camera. The virtual camera setting means may set the positions of the right virtual camera and the left virtual camera so as to be symmetric about the position integrated by the position orientation integrating means, and separated by the distance determined by the virtual cameras distance determination means.
In the present invention, the virtual cameras distance determination means may determine the distance between the right virtual camera and the left virtual camera, based on a parallax between the two pieces of real world image data outputted from the real camera imaging section for a right eye and the real camera imaging section for a left eye.
In the present invention, the virtual cameras distance determination means may calculate the distance between the right virtual camera and the left virtual camera, by using the information about the relative positions and relative orientations of the real camera imaging section for a right eye and the predetermined shooting target with respect to each other, and the information about the relative positions and relative orientations of the real camera imaging section for a left eye and the predetermined shooting target with respect to each other, which are calculated by the position orientation calculation means.
An image display apparatus according to a third relevant embodiment stereoscopically displays a real world image onto which a 3-dimensional virtual object is superimposed, on a screen of a stereoscopic display apparatus capable of providing a stereoscopic view, by using outputs from a real camera imaging section for a right eye and a real camera imaging section for a left eye. The image display apparatus comprises: position orientation calculation means for recognizing a predetermined shooting target in each of pieces of real world image data respectively outputted from the two real camera imaging sections, and calculating position orientation information indicating relative positions and relative orientations of each of the two real camera imaging sections, and the predetermined shooting target with respect to each other; position orientation integrating means for integrating the two pieces of position orientation information calculated by the position orientation calculation means, thereby calculating one piece of position orientation information; virtual camera setting means for determining the position and the orientation of a right virtual camera for generating a virtual image for a right eye, and the position and the orientation of a left virtual camera for generating a virtual image for a left eye, in a predetermined virtual space, based on the one piece of position orientation information integrated by the first position orientation integrating means; right virtual space image generation means for generating a right virtual space image indicating the predetermined virtual space as it is looked at from the right virtual camera; left virtual space image generation means for generating a left virtual space image indicating the predetermined virtual space as it is looked at from the left virtual camera; and display control means for superimposing the right virtual space image onto the piece of real world image data outputted from the real camera imaging section for a right eye, and superimposing the left virtual space image onto the piece of real world image data outputted from the real camera imaging section for a left eye, to output images for a stereoscopic view to the stereoscopic display apparatus.
An image display system according to a third relevant embodiment stereoscopically displays a real world image onto which a 3-dimensional virtual object is superimposed, on a screen of a stereoscopic display apparatus capable of providing a stereoscopic view, by using outputs from a real camera imaging section for a right eye and a real camera imaging section for a left eye. The image display system comprises: a predetermined shooting target; position orientation calculation means for recognizing the predetermined shooting target in each of pieces of real world image data respectively outputted from the two real camera imaging sections, and calculating position orientation information indicating relative positions and relative orientations of each of the two real camera imaging sections, and the predetermined shooting target with respect to each other; position orientation integrating means for integrating the two pieces of position orientation information calculated by the position orientation calculation means, thereby calculating one piece of position orientation information; virtual camera setting means for determining the position and the orientation of a right virtual camera for generating a virtual image for a right eye, and the position and the orientation of a left virtual camera for generating a virtual image for a left eye, in a predetermined virtual space, based on the one piece of position orientation information integrated by the first position orientation integrating means; right virtual space image generation means for generating a right virtual space image indicating the predetermined virtual space as it is looked at from the right virtual camera; left virtual space image generation means for generating a left virtual space image indicating the predetermined virtual space as it is looked at from the left virtual camera; and display control means for superimposing the right virtual space image onto the piece of real world image data outputted from the real camera imaging section for a right eye, and superimposing the left virtual space image onto the piece of real world image data outputted from the real camera imaging section for a left eye, to output images for a stereoscopic view to the stereoscopic display apparatus.
An image display method according to a third relevant embodiment stereoscopically displays a real world image onto which a 3-dimensional virtual object is superimposed, on a screen of a stereoscopic display apparatus capable of providing a stereoscopic view, by using outputs from a real camera imaging section for a right eye and a real camera imaging section for a left eye. The image display method comprises: a position orientation calculation step of recognizing a predetermined shooting target in each of pieces of real world image data respectively outputted from the two real camera imaging sections, and calculating position orientation information indicating relative positions and relative orientations of each of the two real camera imaging sections, and the predetermined shooting target with respect to each other; a position orientation integrating step of integrating the two pieces of position orientation information calculated in the position orientation calculation step, thereby calculating one piece of position orientation information; a virtual camera setting step of determining the position and the orientation of a right virtual camera for generating a virtual image for a right eye, and the position and the orientation of a left virtual camera for generating a virtual image for a left eye, in a predetermined virtual space, based on the one piece of position orientation information integrated in the first position orientation integrating step; a right virtual space image generation step of generating a right virtual space image indicating the predetermined virtual space as it is looked at from the right virtual camera; a left virtual space image generation step of generating a left virtual space image indicating the predetermined virtual space as it is looked at from the left virtual camera; and a display control step of superimposing the right virtual space image onto the piece of real world image data outputted from the real camera imaging section for a right eye, and superimposing the left virtual space image onto the piece of real world image data outputted from the real camera imaging section for a left eye, to output images for a stereoscopic view to the stereoscopic display apparatus.
According to the present invention, it is possible to display a virtual object so as to be normally stereoscopically visible.
These and other objects, features, aspects and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings.